• What?
• Why?
• Where?
• Automatic data pipeline
• Data analysis
• Visualization and Report
• Cross functional collaboration

• Open government data
• Search engine data
• Social media data

• Social media is impactful! [Amusing Ourselves to Death: Public Discourse in the Age of Show Business (1985)]
• Financial resources are sparse
• … and so is our time
• Reproducibility

• API: Twitter/Google/Wikipedia…
• Webpage: Forum, Reviews
• Survey
• Interviews

• https://github.com/ropensci/user2016-tutorial

• http://stat545.com/block022_regular-expression.html

x <- c("here", "is", "P9929AMXT", "a", "P9703AM", "baby", 
    "P0506AM", "example", "P1197AM", "P1271AM")
idx <- grep("(^P)[[:digit:]]+", x)
x[idx]

## [1] "P9929AMXT" "P9703AM"   "P0506AM"   "P1197AM"   "P1271AM"

• If you don't know where to go ……

• If you don't know where to go …

• If you know where to go …

## [1] "English is a crazy language"
## [1] "English muffins"

• When is the best time to tweet
• Who to target
• Product mentioned
• Sentiment score

• From 2016-07-28 to 2016-11-18
• There are 6914 tweets, 1930 original tweets
  
• Products mentioned > 20 times
  • P1197AM
  • P0157AMX
  • P22T73R
  • P28T08R

• Total Tweet Counts

• Tweet and Re-tweet counts by time of the day

library(shiny)
runApp('Rcode/Shiny_NLP')

• No unambiguous yes or no in any country according to current jurisdiction
• So far, court cases (especially in the US) often dealt with commercial interest and often huge masses of data
  • eBay vs. Bidder's Edge
  • AP vs. Meltwater
  • Facebook vs. Pete Warden
  • United States vs. Aaron Swartz

• Encrypt sensitive personal identifiable information
• YOU take all the responsibility for your web scraping work
• If you publish data, do not commit copyright fraud
• If in doubt, ask the author/creator/provider of data for permission
• Consult current jurisdiction

• What is robots.txt?

  "Robots Exclusion Protocol", informal protocol to prohibit web robots from crawling content

• Located in the root directory of a website, e.g. http://baidu.com/robots.txt
• Documents which bot is allowed to crawl which resources (and which not)
• Not a technical barrier, but a sign that asks for compliance
• Syntax in robots.txt

• Scraping etiquette

• Slides and R code can be found here:
  • https://github.com/happyrabbit/BTI2016_12_29
• Future related talk:
  • http://scientistcafe.com